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conducted in an effort to provide recommendations to be applied 
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A statistical chi-square test is conducted on Fleet and Navy wide 
data to determine which ASW combat system material categories are 
most prone to degradation. Additionally, a critical examination of 
the existing WSAT data base is provided with an aim toward 
promoting future statistical analysis. Results of this thesis 
indicate that degradation to weapons delivery systems like torpedo 
tubes and ASROC launchers is statistically more significant than 
the other WSAT test categories. The thesis also recommends new ways 
ito adapt the existing WSAT data base to conduct more informative 
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I. INTRODUCTION 


A. WEAPONS SYSTEM ACCURACY TRIAL (WSAT) BACKGROUND 
1. History 

The history of the Anti-Submarine Warfare (ASW) Weapon 
System Test — for surface ships dates back to post World 
War II studies of ineffective torpedoes, sensor alignment 
errors, and fire control accuracy problems. Presentation 
materials obtained from the Undersea Warfare Museum in 
Keyport, Wa. indicate initial systems level tests conducted on 
Dabob Bay, Washington as early as 1959 led to _ the 
establishment of WSAT tests and began the genesis of the 
surface ship ASW test program. Presently, WSAT is a total 
system test which demonstrates the ability of a ship’s ASW 
combat system to pass prescribed equipment performance 


standards tests. The total test concept is shown in Figure 1. 


WSAT — A TOTAL SYSTEM TEST 


TARGET 


SENSORS 
(SONAR, EM LOG, GYRO/SINS) 


SME UNDERWATER 
CTORPEDO TUBES/ASROC) FIRE CONTROL 





Figure 1. WSAT Overview 


2. Responsibilities 

Commander Naval Seas Systems Command (NAVSEA O6UT) has 
overall responsibility for WSAT and is the activity 
responsible for surface ship ASW system certification and 
recertification. The WSAT results are used by NAVSEA as a 
basis for granting this certification. ASW certification is 
required on every surface ship ASW combat system subsequent to 
new construction and commissioning. Similarly, ASW system 
recertification 1s required on existing surface ships which 
hold an ASW required operational capability upon exiting 
regular overhaul (ROH) or upon receiving a major ASW system 
suite upgrade in some other industrial period. Commander 
Naval Underwater Systems Center (NUSC), Newport, Rhode Island 
is designated as the WSAT program technical direction agent 
(TDA) responsible for all WSAT test and analysis procedures 
and documentation. 

The WSAT organization supports both Atlantic and 
Pacific Fleet units. Surface ship WSAT inspections on the 
East coast are conducted by NUSC technicians at either the 
Atlantic Fleet Weapons Training Facility (AFWTF) 3-D 
underwater range near St. Croix with dockside tests at Naval 
Station Roosevelt Roads, Puerto Rico or at the Atlantic 
Undersea Test and Evaluation Center (AUTEC) 3-D range in the 
Bahamas with dockside tests done in Port Everglades, Florida. 


Surface ship WSAT inspections on the West coast are 


accomplished by technicians from Naval Undersea Warfare 
Engineering Station (NUWES) detachment San Diego, California 
on the San Clemente Island 3-D Underwater Range (SCIUR) with 
dockside tests at Naval Station San Diego or Naval Station 
Long Beach. In the Hawaii area, inspections are done by NUWES 
detachment Hawaii on the Barking Sands 3-D Tactical Undersea 
Range (BARSTUR) with dockside tests done at Naval Station 
Pearl Harbor. 
3. Objectives 
NAVSEA Publication OD 40087 (Ref. 1:pp. 1-1], the ASW 
WSAT program manual defines the objectives of the WSAT program 
as follows: 
e To ensure each surface ship ASW system reaches the fleet 
in satisfactory condition and is able to meet its assigned 
operating capabilities. 


e To identify system defects prior to the expiration of 
contractual industrial warranties. 


e To develop and analyze data on system specifications and 
tolerances and refine the same. 


e To identify the appropriate command or agency responsible 
when system discrepancies degrade performance. 


e To promote meaningful system improvement programs. 


4. Inspection Sequence Of Events 
When the Type Commander requires a WSAT conducted on 
one of his ships it is accomplished via request through the 
Fleet Commander scheduling system in conjunction with the 


responsible WSAT agency. The responsible activity then 


assembles a stock test plan for the ship based on its 
configuration. Accordingly, the scope of the inspection 
varies, based not only on the ASW suite installed, but also on 
the inspectors available at inspection time. The WSAT 
consists of four inspection phases culminated with exercise 
weapons firings on an instrumented 3-D range to evaluate total 
ASW system dynamic performance. The WSAT consists of both 
dockside and underway testing, evaluation, and analysis. A 
summary of each WSAT inspection phase is provided as follows: 
e Phase I (5-7 Days Dockside) 
1. Arrival conference and equipment set up. 
2. ASW fire control operability and accuracy checks. 
3. Internal sonar alignment checks. 
4. ASW system interface tests and alignments. 
5. Gyrocompass and inertial navigation error tests. 
6. Onload of exercise weapons and range instrumentation. 
e Phase II (1 Day Dockside) 
1. Fire control and torpedo tube interface tests. 
2. Tube firing voltage and launch pressure tests. 
3. Dummy MK-46 torpedo firings on each tube. 
4. Measurement of weapon exit velocity. 
5. Weapon recovery, inspection, and preset verifications. 
e Phase III (1-2 Days Underway) 
1. 3-D range Sensor Accuracy Test (SAT) 
2. Calibration of electromagnetic log. 


3. Gyroscope and inertial navigation heading checks. 


e Phase IV (1 Day Underway) 
1. Total system end-to-end test on 3-D range. 
2. MK-46 over the side exercise torpedo firing. 
3. Exercise ASW rocket (ASROC) firing from each launcher. 


4. Helicopter exercise torpedo firing (if so equipped). 


A summary of WSAT exercise firings is provided in Figure 2. 


NEW REGULAR OVERHAUL 
CONSTRUCTION OR 4 YEAR SRA 


SVTT only 


SVTT and 
LAMPS MK Il 


SVTT and 
ASROC 


SVTT, ASROC 
and LAMPS 
MK Il 


SVTT, DUAL 
ASROC LAUNCHERS 


SVTT, DUAL 
ASROC LAUNCHER 
and LAMPS MK III 





Figure 2. WSAT Firing Exercise Summary 


Upon completion of the WSAT , test results are promulgated 
in the following manner: 
® WSAT quicklook message report (24 Hours) 
1. Identifies all major, uncorrected discrepancies. 
e WSAT final message report (10 Days) 
1. Identifies all remaining uncorrected deficiencies. 
2. recemnonae corrective action on problems. 


3. Provides NAVSEA recommendation for certification. 


5. Record Keeping 

After the WSAT final report message is received by the 
cognizant inspection activity it is tracked until major 
discrepancies prohibiting certification are cleared (if 
certification is not initially achieved). At that time NAVSEA 
issues a certification statement for the ship concerned. Then 
the inspecting activity files the hard copy of the WSAT final 
report message in its respective technical library. NAVSEA 
(O6UT) then utilizes the ASW test program data base for 
storage of pertinent WSAT information. NAVSEA, NUSC, and 
NUWES detachments are the holders and users of this 
microcomputer based data base. It will be described and 


evaluated in Chapter III of this analysis. 


B. WSAT PROBLEMS 
7. Cost 

In recent years funding limits have precluded 
accomplishment of WSAT on all required ships, particularly 
those older ships exiting an industrial environment. 
Interviews with officials at NAVSEA have clarified the policy 
that all ships with ASW missions will receive WSAT or some 
reduced scope facsimile thereof rather than inspect just a few 
ships with a complete WSAT. Currently, AEGIS and BURKE class 
new construction and SPRUANCE class ASW suite conversions are 
receiving priority for funding. As a result of funding 
problems, older, 
less ASW oriented, 
ships such as non- 
AEGIS class 
cruisers and PERRY 
class guided 
missile frigates 
may not be funded 
for recertification 
inspections as they SAT - SURFACE SHIP (1 DAY) 
exit regular Se ene nye! 
overhaul or other 


such industrial 





availabilities. Figure 3. West Coast Cost Data (Thousands) 


This is of concern since most of these ships are tentatively 
scheduled to remain on the active duty roster through the year 
2000. Figure 3 provides fiscal year 1992 budget estimates of 
West coast WSAT costs based on required inspections forecast 
in the ASW: test program data base. Force downsizing will be 
providing relief with the ongoing decommissioning of all KNOX 
class frigates as well as the COONTZ and ADAMS class guided 
missile destroyers. As for the non-AEGIS cruisers of the 
LEAHY and BELKNAP class which will remain on active duty in 
the near future, the April 1992 Navy Times [Ref. 2:p. 4] 
outlined Washington Navy proposals to possibly remove or at 
least deactivate sonar and ASW weapons systems from these 
ships. This effort will cut operations and maintenance costs 
and take a step toward eliminating obsolete systems from the 
fleet. It would also free up additional training facilities 
ashore and provide cost savings in the manpower area. The 
overall result with respect to ASW test programs would be 
fewer ships to certify. 
2. Redundant Inspections 

An additional problem effecting complete funding of 
required WSAT inspections is that of redundant inspections. 
The current impetus at NAVSEA is to eliminate redundant 
inspections, or at least to consolidate testing where possible 
to promote greater fiscal efficiency. An example of this is 


the AN/SQS-89(V) surface ship ASW suite certification. This 





initial acceptance certification, which contains many of the 
same testing elements as WSAT, pertains to new. ship 
construction programs and is intended to ensure specifications 
are met satisfactorily. This program is an important part of 
acquisition quality assurance policies in the Navy’s dealings 
with outside vendors. Interview with Mr. Bob Devon of 
NAVSEACENPAC indicates that early timing of this certification 
with respect to crew training and arrival onboard from 
precommissioning sites may preclude substitution for WSAT in 
whole. However, some portions of the testing are candidates 
for consolidation in WSAT testing. Another example of this is 
the ASW Systems Qualification Trials (SQT), a part of the 
greater Combat System Shipboard Qualification Trials (CSSQT). 
Defined under NAVSEAINST 9093.1A, CSSQT encompasses the entire 
weapons suite and is heavily oriented to both operations and 
maintenance training. Peikenouahy a favorite of the fleet due to 
the expansive scope of services, exercise shots, and training 
opportunities, NAVSEA has emphasized its responsibility for 
material readiness and not for training. Although no cost 
figures were available on CSSQT or ASW SQT, it is certain that 
costs are several times that of WSAT alone. Despite the fact 
that CSSQT and ASW SQT meet all requirements for WSAT, the 
certification processed is subject to competing demands in 
other combat systems areas. Likewise, WSAT inspection 
scheduling and budget controls at NAVSEA (06UT) are 


relinquished under CSSQT cognizant authorities. 


As of January 1992, CSSQT and ASW SQT are unfunded for the 
foreseeable future with the exception of lead ships in a new 
class such as the BURKE class guided missile destroyer. 
3. Reduced Scope Inspections 

In an effort to provide a WSAT for every ship when 
due, the current effort is to develop a reduced scope 
inspection which will meet certification requirements. 
Specifically, the requirement for recertification after ROH or 
DSRA is of interest because statistically these were the ship 
inspections left unscheduled due to funding constraints. The 
first endeavor for a reduced scope WSAT inspection was the 
Surface ASW System Test (SAST). The SAST Program Manual 
(Ref. 3:p. 1-2] outlines the scope of the inspection. Figure 
4, taken from that manual, displays the proposed relationship 


of the SAST program to the ship operational life cycle. 


ASW COMBAT SYSTEM 
CERTIFICATION 
DPERATIONWAL 
Pron] [eerwe wananrcy] == [Com] 


"AS REQUIRED AS REQUIRED 


ASW COMBAT SYSTEM 
CERT. VALIDATION 
a 

OPERATIONAL 





Figure 4. Ship Life Cycle (SAST) 
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The first SAST inspection was accomplished in January 1992 at 
Long Beach Naval Station, California aboard the USS PAUL F. 
FOSTER (DD-964). The SAST inspection was designed to cost 
approximately one third that of a WSAT and provide a 
reasonable assurance that the ASW combat system is ready for 
underway operations and meets requisite performance 
specifications. Although the four phase SAST inspection is 
similar to WSAT on paper, some significant differences exist. 
First, a shift of emphasis from outside source technician 
testing to ship’s force requires that the ship complete Phase 
I dockside testing prior to the arrival of NUWES/NUSC ASW 
system experts in phase II. As a result, ship’s force PMS 
checks now form the basis for continued testing. This 
assessment is performed by the SAST team leader. This concept 
removes the necessity of a WSAT team from being on travel for 
the usual WSAT phase I (5-7 days) dockside tests. 
Unfortunately, it shifts the foundation for ASW certification 
to the command involved without regard to current ship manning 
and training levels subsequent to an industrial availability 
when they are historically under strength. As a result some 
material categories could receive little or no attention by 
the more qualified NUSC/NUWES engineers in phase II, based on 
erroneous testing by ships force early in phase Ul. 
Additionally, the phase IV WSAT 3-D range firing exercises are 
eliminated. This concept eradicates the total system end-to- 


end test deemed important by the fleet as well as test 
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engineers. Also, it is unclear if anywhere in SAST an effort 
is made to scientifically determine where to concentrate 
scarce inspection dollars based on material discrepancy trends 
or histories. The latest endeavor at NAVSEA in restructuring 
WSAT, currently in the planning stage only, is referred to as 
the Consolidated ASW Test (CAT) program. The current surface 


ship ASW test plan POA&M is shown in Figure 5. 


SURFACE SHIP ASW COMBAT SYSTEM 
CERT PROGRAM POA&M 


Establish mimimum trial requirements 


Define dockside requirements (Pt 1, Pt 2) 
Underway requirements 


Prepare typical PMS test pkg (for dockside Pt 1) 
Review PMS test pkg (TYCOMs, NAVSEA) 

Prelim test plan & daily schedule 

Review test plan/schedule (by TYCOMs, NAVSEA) 
Final/approved test plan/schedule 

Schedule first ship for CAT trials 

Brief first ship on CAT 

Conduct first CAT trials 


92/ Monitor, evaluate effectiveness of CAT 
92 Propose revisions and improvements 





Figure 5. ASW Test Program POA&M 


dee 





The goal of the CAT program appears similar to SAST in that it 
will define the post-shipyard ASW combat system certification 
program for surface ships. The program will meet OPNAV 
requirements to consolidate redundant inspections as well as 
address fleet concerns for collateral training opportunities 
alongside NUSC/NUWES technicians in both operations and 
maintenance. The process is currently under Total Quality 
Leadership (TQL) review to provide quality performance 
indicators which measure effectiveness and ensure ongoing 
program improvement. The lessons learned from the only SAST 
inspection performed will be incorporated by increasing 
NUSC/NUWES supervision during phase I dockside testing. As in 
SAST, increased reliance on ships force personnel in 
conducting selected maintenance checks will be utilized. This 
program will attempt to incorporate the existing ship PMS 
schedule with results from applicable shipyard testing to 
reduce or eliminate redundant and excessive testing. However, 
ie program core testing requirements do not specify any 
analysis of systems discrepancy histories in regard to 
constructing individual ship test packages. Similar to SAST, 
elimination of 3-D range torpedo exercises are sacrificed in 


lieu of simulated ASROC, OTS, and LAMPS exercises. 


Hs, 


4. Ramifications and Concerns 

The overall thrust of the new ASW systems test program 
appears to make the best of a bad situation with regard to 
budget shortfalls. Funding of testing for new construction 
BURKE class guided missile destroyers will undoubtedly be 
preserved, the problem lies with recertifying the aging 
SPRUANCE and PERRY class ships. Likewise, the AEGIS cruisers 
of the TICONDEROGA class will also require some type of 
recertification inspection as they progress through their life 
cycle. The last ship of this class was commissioned in July 
1992. The shifting of testing prerequisites to ships force as 
a prelude to ASW certification will pose another scheduling 
problem for afloat commands. This problem can be minimized by 
adequate involvement of the type commander and NAVSEA when 
planning ship overhauls. Additional efforts on the part of 
the ship will be Penis to ensure a core of sufficiently 
trained technicians are onboard at-certification time. This 
will minimize possible compromise of initial phase dockside 
testing by unqualified technicians under pressure to meet 
schedule deadlines. A resounding need for ongoing statistical 
analysis in material category reliability is called for to 
prevent planned testing initiatives from risking overlook of 


problematic areas in construction of a ship test package. 
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C. THESIS OBJECTIVES 
1. Material Discrepancy Data Analysis 
With the substantial data available on WSAT ASW combat 
system material discrepancies, the first objective of this 
thesis is to perform a categorical data analysis to reveal 
which material categories are most prone to failure. This 
would refute an implicit assumption that all material 
categories are uniformly degraded and that reduced scope 
inspections need not statistically consider system material 
histories during program conception. 
2. WSAT Data Base Examination 
The second objective of this thesis will be a critical 
examination of the existing WSAT data base. This examination 
will include recommendations for future modifications to the 
data base; primarily with respect to the deletion and addition 
of specific data fields. These recommendations will be biased 
toward what will be required to make the data base a viable 


tool in future statistical analysis. 
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II. WEAPON SYSTEM ACCURACY TRIAL (WSAT) DATA 


A. DATA COLLECTION 
1. Experience Tour 
During the six week ASW curriculum experience tour in 
quarter 5 of 8, several invaluable point of contacts were made 
to facilitate WSAT thesis data collection. Mr. Edward Biery 
of NUWES detachment Keyport, WA. (code 56) was sponsor of the 
experience tour. It was his office from which most background 
information on the WSAT program was obtained. That office 
arranged a visit to NAVSEA (06UT), Washington, D.C. to meet 
senior civilian engineers in the ASW test program as well as 
STGCS(SW) W.J. Vick, developer of the NAVSEA ASW test program 
Gata base. During this visit NAVSEA policies and goals with 
respect to the WSAT program were discussed and clarified. 
2. Data Sources 
The actual data for use in this analysis was obtained 
from various subordinate activities to NAVSEA. Specifically, 
the data was obtained in naval message format from the 
following activities: 
e Pacific Fleet Data 


1. NUWES detachment, San Diego, CA. (code 906), Mr. Alex 
Rios (Head ASW Analysis Division). 


2. NUWES detachment, Hawaii (code 90), Mr. Adolph Neumann 
(Director Pacer). 


INS 


e Atlantic Fleet Data 


1. NUSC Newport, R.I. (code 38), Mr. John Peters (ASW 
Test Analyst) 


A summary of the WSAT data received for this analysis is 


provided in Figure 6. 


WSAT DATA SOURCES 
BY SHIP TYPE 


SHIP TYPE BY CLASS NUMBER OF SHIPS TESTED 
GARCIA (FF) 

KNOX (FF) 

USCG (WHEC) 
BROOKE (FFG) 
PERRY (FFG) 
SPRUANCE (DD) 
KIDD (DDG) 

BURKE (DDG) 
LEAHY (CG) 
TICONDEROGA (CG) 


TOTAL NUMBER 
OF. SHIPS TESTED 


SURFACE SHIP WSAT DATA (5 YEARS) 





Figure 6. WSAT Data Sources 
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B. DATA ORGANIZATION 
1. Data Specifics 
A total of 64 WSAT final inspection report messages 
covering the last five years were obtained from cognizant 
inspection activities for use in this analysis. The data 
represents both Pacific and Atlantic fleet surface assets 
which have ASW as one of the command primary warfare areas. 
The last five years of data was chosen for this analysis: 

1. To minimize interruption of business at the various 
inspection activities who were called on to locate, 
reproduce, and ship the data. 

2. To minimize wasteful receipt of older data which is 


representative of ships scheduled for upcoming 
decommissioning. 


The 64 final report messages contain a total of 595 inspection 
discrepancies. These data elements chosen for analysis are a 
representative population sample of all WSAT discrepancies. 
Each data element represents a failed maintenance inspection 
or operational check. For purposes of the categorical data 
analysis to be discussed in Chapter IV, each check is the 
result of a failed binomial experiment. The sampled data was 
further assigned to one of five specific material categories. 
The categories were not chosen arbitrarily, but rather were 
chosen per the prescribed categorical reporting format of the 
final report message described in NAVSEA OD 40087. With all 


inspection reports describing both major and minor 


18 





discrepancies in this manner, aggregation of all data elements 
into appropriate categories became a less arduous 
administrative task in bookkeeping. 
2. Data Categories 
To clarify which material discrepancies belong to 
specific categories, a breakdown iS summarized as follows: 
e Category I (Sonar Systems) 
1. Hull mounted sonar systems 
2. Towed array sonar systems 
3. Signal processing systems 
®e Category II (ASW Aviation Systems) 
1. Sonobuoy systems 
2. ASW helicopter data link systems 
3. Torpedo loading, arming, and presetter systems 
e Category III (Weapons Delivery Systems) 


1. Surface vessel torpedo tubes, all makes’ and 
modifications 


2. Asroc box launcher systems 
3. Asroc guided missile launcher systems 
4. Asroc vertical launch systems 

e Category IV (ASW Fire Control Systems) 
1. The Mk-114 analog fire control system 
2. The Mk-116 digital fire control system 


3. The USCG Mk-309 fire control system 


dee) 


e Category V (ASW Sensor Subsystems) 
1. The ship electromagnetic log 
2. Ship gyroscope and inertial navigation systems 
3. ASW related command decision systems (CDS/NTDS) 
4. ASW related gunfire control radars and systems 


5. ASW related consoles and repeaters 


C. DATA SUMMARY PRESENTATION 

For presentation of the data by categories bar graphs were 
chosen for relative comparisons between material categories. 
For this analysis the data 1s broken down by Atlantic and 


Pacific fleet and also shown Navy wide in Figures 7, 8, and 9. 


ATLANTIC FLEET WSAT 
DISCREPANCIES BY CATEGORY 


SONAR SYS. AVIATION SY3. WEAPONS SYS. KC. SY3. SENSOR SUBSYS. 
30 70 67 39 


SURFACE SHIP WSAT DATA (5 YEARS) 





Figure 7. Atlantic Fleet WSAT Data 


20 


PACIFIC FLEET WSAT 
DISCREPANCIES BY CATEGORY 


0 SONAR SYS. AVIATION SYS. WEAPONS SYS. F.C. SYS. SENSOR SUBSYS. 
£ 62 26 93 63 90 


SURFACE SHIP WSAT DATA (5 YEARS) 





Figure 8. Pacific Fleet WSAT Data 
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NAVY WIDE WSAT 
DISCREPANCIES BY CATEGORY 


AVIATION SYS. WEAPONS SYS. F.C. SYS. SENSOR SUBSYS. 
56 163 1380 129 


SURFACE SHIP WSAT DATA (5 YEARS) 





Figure 9. Navy Wide WSAT Data 





III. THE WSAT DATA BASE 


A. DATA BASE RESPONSIBILITY 

The WSAT data base 1S managed by the NAVSEA ASW test 
program office (NAVSEA O6UT). It is the product of efforts by 
STGCS(SW) W.J. Vick, USN. It was designed to be an all 
inclusive information system containing relevant historical 
information concerning ASW testing conducted by NAVSEA on both 
surface ships and submarines. Additionally, it is intended to 
be used as an aid in forecasting future testing requirements 
based on long range ship repair and conversion plans. The 
information in the centralized data base is compiled from many 
different sources. The data base iS currently managed by Mr. 
Peter Karounos (NAVSEA O6UT) who is assisted by STGCS(SW) J. 
Brown, USN. As a centralized data base, NAVSEA retains 


responsibilities for both data integrity and security. 


B. ORGANIZATION 

The latest printed edition of the WSAT ASW test data base 
is dated 16 September 1991 and is organized into two parts. 
Part one is information relevant to surface ships. Part two 
is information relevant to submarines. Further discussion 
will be limited to the surface ship section of the data base 
in accordance with the scope of this analysis. As a 


microcomputer based software data base management system 
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written on the commercially available PFS:Professional File 
Version 2.02, the data base is easily learned and manipulated. 

G. Hanson and J. Hanson in Database Management and Design 
(Ref. 4:pp 15] describes a microcomputer data base management 
system aS a system that demonstrates query flexibility and 
ease of data base creation and maintenance. This package 
meets those prerequisites. The surface ship data base 
currently comprise 51 data fields of various types (ie: date, 
numeric, alphanumeric, etc...). Figure 10 displays the 


relevant data fields being used. 


SURF Enter instructions for printing mailing labels. Page 1 of |2 
Fl-Help F2-Label F3-Edit F4-Quick Entry F10~continue 
eS cc TS ne a a 


HULL: : uIc: 
CLASS: 


TYCOM: PAC: : : SQN: 


FY: QTR: : SUPPORT: 


AVAIL: : END YARD: 


LAST WSAT: 
SCOT: 


CSSQT: 


SAT RPT: SAT DATE: 


SAST: $ PHONE: 
SAST LTR: : #mDEF: 


WSAT: 

QUICK LOOK: 

FINAL REPORT: MINOR: 
NOTES: 


WITHHOLD LTR: 
CERT LTR: 
REMARKS : 





Figure 10. WSAT Data Base Fields 
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C. CRITICAL EXAMINATION 

PFS:Professional File is a flexible software application 
package which is very Similar to Data Base III. The 
comprehensive use of pull down menus and on line help 
functions make it quite adaptive to the novice user, yet it 
also has many short cut features suited to the advanced 
operator. Although the software system is more than adequate 
for the job, the data base itself has several aspects needing 
revision to ensure continued usefulness. Critical examination 
indicates that while a basis for corporate information system 
Management is present, aceite Cede 8 data fields in the data 
base are either obsolete or unnecessary. This statement is 
substantiated by the appalling lack of data in these fields 
and the insignificance of the information in others. A 
complete printed report of the entire data base requires 4 
pages to intelligently convey all information on each ship. 
It 1s obvious upon examination that the data base is primarily 
an administrative aid to track ASW testing, scheduling, and 
associated correspondence. Unfortunately, the usefulness of 
the data base in identifying material discrepancy trends is 
limited in the current format. Although the number of 
discrepancies are listed for each ship inspection, it fails to 
identify equipment specifics or even the associated material 
categories. This information will be statistically 


Significant in planning future inspections oor _ other 
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certification procedures. This statement is even more true in 
a time of budget shortfalls and constraints, which appear to 
be a permanent fixture associated with the conclusion of the 
cold war. 

As future certification procedures evolve, probably based 
on cost overhead, it would be imprudent to ignore material 
trends where ween resulting from non-testing are too great. 
The problem of manpower in maintaining the ASW test data base 
might be addressed by changing from a centralized data base 
Management system to a distributed system connected via modem. 
This would spread the chores of data entry to the requisite 
inspection activities and put NAVSEA in a more supervisory 
role. The network could conceivably consist of only 5 sites 
which is easily manageable. With the unclassified nature of 
the data base and the use of commercial phone lines, this 
concept iS certainly economically feasible. Spéc Mite 
recommendations on data base field deletions and additions 


will be included in Chapter VI of this study. 
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IV. ANALYSIS OF DATA 


A. ANALYSIS ASSUMPTIONS 

As the scope of the ASW test and certification progran, 
specifically WSAT, undergoes dynamic changes to meet funding 
restrictions, the question of which material categories 
historically demonstrate the most discrepancies should be 
examined. If the WSAT core structure inspection is reduced 
without regard to this, then it is an implicit assumption that 
material degradation is homogenous throughout the five 
material categories in the WSAT. Although WSAT describes 
inspection discrepancies as either major or minor in nature, 
this analysis will weigh all discrepancies equally for 
conducting the multinomial experiment. Accordingly our N = 
595 data points will be individually assigned to one of five 
material categories. The variable p, will represent the 
probability that any individual discrepancy belongs in 


material category i (i=1..... By. 


A. NULL AND ALTERNATE HYPOTHESIS 

In order to conduct a statistical test on the data, a 
specific hypothesis to be refuted must be defined. For this 
analysis the null hypothesis will be that any particular 


observation is equally likely to fall in any one of 


aM 


the five WSAT material categories described in chapter II, 
that 1S DP,=P,=P;=P-=Ps;- The alternate hypothesis to be 
substantiated via testing is that at least one category 
proportion is not equal to the others. Equation 1 summarizes 


the null hypothesis: 


NULL HYPOTHESIS (1) 
H, : Py = Py = P3 = Py = Ps = -20 


Equation 2 summarizes the alternate hypothesis: 


ALTERNATIVE HYPOTHESIS 2) 
H, : At least one p, # .20 


B. STATISTICAL TESTING OF HYPOTHESIS 

A standard test procedure for this hypothesis is described 
in Devore, Probability And Statistics For Engineering And The 
Sciences [Ref. 5:pp 556-559}. The name of this procedure is 
the chi-squared goodness-of-fit test for categorical data. 
The null hypothesis is rejected in favor of the alternative 
hypothesis if a test statistic is too large. The test 
statistic is labeled a x’ (chi-square) statistic and is 


defined in Equation 3 


5 
x =) ee (3) 


28 


where n, is the number of the total N = 595 observations that 
fall into each material Gaceger at) (1=19273,4,5) found in 
Chapter II, Figure 9. The variable e, is the expected number 
of the 595 observations for each category 1 as calculated in 


Equation 4. 


e. = p(N) = (.20)(595) = 119 (4) 


When the null hypothesis is true, the observed number of 
category discrepancies should be close to the corresponding 
expected values and the test statistic should be small. 
Provided that the expected value, e,, 1s greater than or equal 
to 5 for each category, the test statistic has a chi-squared 
Gistribution with vy = 5-1 = 4 degrees of freedom. The 
decision to accept or reject the null hypothesis will be 
accomplished by tig the computed value of the test 
statistic to an appropriate critical value of the chi-squared 
distribution, which is tabulated and available in most 
statistics text books. The critical value for the test 
procedure will be determined by choosing a significance level, 
a , for the hypothesis test. The value of a is the 
probability of rejecting the null hypothesis when it is true. 
The critical value is the 1-a percentile point of the chi- 


square distribution with 4 degrees of freedom. For a =.01 and 


y = 4 the tabulated critical value is 13.277. 


2g 


This means we reject our null hypothesis if the calculated 
test statistic is greater than 13.277. Equations 5 through 7 


Summarize the Navy wide chi-square test respectively. 


Reject H, (p, =.20) if 


2 2 
X 2X ey 


(5) 


The x? test statistic defined by equation (3) 
2 _ (117-119) , (56-119)* | (163-119)” 
a ee eee eee 
119 119 119 (6) 
-_ as a, 2 
, (1302119)? | (295 6IB)" we cae, 
119 119 


Lhe tabular distribution critical value Is 
og = 13.277 
as a result, since 
17 S"51 Ss eS ete ~Reject H, 


(7) 


The conclusion, based on this test procedure, is to reject the 
null hypothesis of homogenous material category degradation in 
favor of the alternate hypothesis. There iS very strong 
statistical evidence indicating significant differences in 
Navy wide WSAT material category reliability to support the 
alternative hypothesis. 

The next step in the analysis will be to determine if the 
Atlantic Fleet data refutes the same homogenous null 


hypothesis. Using observed discrepancy values from Chapter 
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II, Figure 7, the pertinent variables are summarized in 


Equation 8 through 10. 


N = 261 p, = .20 e, = (.20)(261) =52.2 (8) 


The x? test statistic is 
pee tee? 2a 0552.2)* (70-52.2)° 


S22 Sp 52.2 (9) 
e 2 2 2 
MCT), (O9552.2)hewrane, 
52.2 OP 
x7=23.20 2 x7 9,,=13.277 --------- Reject H, (10) 


Accordingly, testing of the Atlantic Fleet data also supports 
rejecting the null hypothesis in favor of the alternative 
hypothesis that at least one p, is not equal to .20 

The last step in the test will be to check that the 
Pacific Fleet data also refutes the null hypothesis. Using 
the data from Chapter II, Figure 8 the variables for the test 


are summarized in Equations 11 through 13. 


N = 334 p, = .20 e. =(.20)(334) = 66.8 (11) 


The ? test statistic is 
oe (62-66.8)* , (26-66.8)" | (93-66.8)° 


66.8 66.8 66.8 (12) 
= 2 = 2 

, (63-66.8) , (90-66.8)" _ 43 95 
66.8 66.8 


2 


X” = 43.82uzupiieny | Sale Tete ~Reject H, (13) 


Like the Navy wide data and the Atlantic Fleet data, the 
Pacific Fleet data also strongly supports rejecting the null 
hypothesis in favor of the alternative hypoyhesis. 

Having tested the data in the aforementioned fashion, it 
1S now an accurate statement to say that a homogenous 
degradation in the five material categories of WSAT is false. 
Chapter V will address specific recommendations on which 
category to focus attention on and which categories are least 


prone to failure. 


32 








V. CONCLUSIONS AND RECOMMENDATIONS 


A. WSAT ANALYSIS 
1. Conclusions 
The analysis of the last five years of WSAT data 
conducted in Chapter IV strongly indicates the variation in 
degradation among the five material categories which comprise 
WSAT. Figures 11 through 13 summarize the experiment results 
for the Atlantic Fleet, Pacific Fleet, and Navy wide data 


respectively. 


FREQUENCY DISTRIBUTION 
PACIFIC FLEET OBSERVATIONS 


FREQUENCY DISTRIBUTION 
ATLANTIC FLEET OBSERVATIONS 


CJ OBSERVED 
~ EXPECTED 


CL) OBSERVED 
+ EXPECTED 


I....SONAR_SYS .»--SONAR_SYS 


11... AVIATION SYS .. AVIATION SYS 
(ii. WEAPQNS CYS WRAPONS SYS 


syalPalo aki 





IV...F.C..SYS _ 





V....SEMSOR SUBSYS 
VV Vv 


»-» SENSOR_SUBSYS 


(5 YEARS WSAT DATA) 


(5 YEARS WSAT DATA) 





Figure 11, Atlantic Fleet Results Figure 12, Pacific Fleet Results 
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FREQUENCY DISTRIBUTION 
NAVY WIDE OBSERVATIONS 


[_] OBSERVED 
+ EXPECTED 


..-SONAR SYS 
.. AVIATION SYS 
..WEAPONS SYS 
ther \ 5 


.... SENSOR SURSYS 
UIT IV V 


(5 YEARS WSAT DATA) 





Figure 13. Navy Wide Results 
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The Navy wide material categories rank from least prone to 
failure to most prone to failure as follows: 
e NAVY WIDE RELIABILITY RANKING 
1. Aviation Systems (Cat IT) 
2. Sonar Systems (Cat I) 
3. Sensor Subsystems (Cat V) 
4. Fire Control Systems (Cat IV) 


5. Weapons Delivery Systems (Cat IIT) 


The discrepancy most common in the least reliable category was 
undeniably found to be in torpedo launcher systems. Improper 
tube firing voltages, firing pressures, and interface failures 
were prolific in the 163 data points in this category. This 
is not surprising considering the fact that most torpedo 
launcher systems have at least partial exposure directly to an 
ocean environment with detrimental effects on electrical and 
mechanical components. Figure 14 provides for direct 
comparison between Atlantic and Pacific Fleet observed 
discrepancies. In 3 of 5 categories the Pacific Fleet ASW 
systems demonstrate a greater propensity for failure than the 
Atlantic Fleet. The largest variation found was in category 
V (Sensor Subsystems) where the Pacific Fleet showed 
Significantly higher numbers of discrepancies than its 
contemporary (90 vice 39). Subsequent reexamination of the 
data revealed a prevalence of discrepancies in NTDS/CDS 


software on Pacific Fleet Ticonderoga class guided missile 
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cruisers. These most recently delivered ships contain the 
most current software versions which apparently are replete 
with certification discrepancies. The other differences in 
Fleet data are small in comparison but have no apparent 
rational explanation. One would expect that uniform 
technician training standards and maintenance procedures would 
minimize this variance. The only other possible explanation 
might be a function of East Coast and West Coast Test Activity 


inspector personalities and background experience. 


FREQUENCY DISTRIBUTION COMPARISON 
BETWEEN ATLANTIC AND PACIFIC FLEETS 


(J ATLANTIC FLEET 
A PACIFIC FLEET 


1... SONAR SYS 
I]. AVIATION SYS 
4 I. WEAPONS SYS 
|Z W..F.C. SYS 


2 -¥....SENSOR SUBSYS 


© 
k 


7A 
Y 
A 
Z 
LZ 
Y 
g 
ZY 
Y 
A 
A 
Z 
Z 
VA 

4 
| 


(S YEARS WSAT DATA) 





Figure 14. Fleet Data Comparison Summary 
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2. Recommendations 
The following recommendations apply to the future of 
the ASW Test Program: 


1. Focus future inspection packages with emphasis on torpedo 
tube and ASROC launcher validation by experienced 
technicians. 


2. Establish revised, precise tactical digital standards for 
certification of software based systems. 


3. If 3-D range firing exercises must be eliminated, conduct 
a comprehensive study of past range data to determine 
failure cause and results. Incorporate these findings in 
test program development. 


4. Include any available ASW PACER data in development or 
assessment of a ship ASW certification. 


5. Conduct ongoing statistical analysis of ASW combat 
systems discrepancy trends. The CASREP data base, 
maintained at the TYCOM level, is an excellent source of 
current Fleet system problems. 


6. Incorporate future WSAT (or equivalent) data into ongoing 
statistical analysis. 


7. Conduct a study to compare WSAT historical trend results 
with current Fleet equipment reliability prediction 
programs such as the Trouble Systems Process developed by 
eiero Corporation. 

8. Conduct a comparative study of ASW technical training 


standards between Fleet training centers with a goal of 
uniformity. 


B. WSAT DATA BASE ANALYSIS 
1. Conclusions 
The critique of the existing WSAT data base in Chapter 


II found a comprehensive, centralized data base requiring only 
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minor modification. The key issues are to examine the 
implementation of a distributed network data base and 
subsequent modification of data base fields to promote 
statistical analysis. 
2. Recommendations 

The following recommendations apply to data base field 

structure: 
e Fields Recommended For Deletion 
1. GRP/SQN/PORT: All of these fields are variable over a 

ships life. They are available elsewhere and actually 

provide little information relative to the ASW Test 

Program. 


2. CSSQT: This is no longer a funded program. 


3. SCOT: This field contains minimal existing data which 
attests to its invalidation. 


4. SAST/SAST LTR: Only one inspection was conducted under 
this program prior to discontinuation. 


5. TDA/PHONE #: This field also has no existing data. 
Test agency phone numbers are variable as shore 
commands evolve and are available elsewhere. 


6. REMARKS: This field exhibits minimal existing data 
which attests to its invalidation. 


e Fields Recommended For Addition 


1. A material category discrepancy breakdown field which 
would provide an adequate means of recording observed 
discrepancies in an orderly fashion and promote 
statistical analysis. The following numeric field 
format would suffice: --/--/--/--/--. 


Sis 
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